# Rethinking FID: Towards a Better Evaluation Metric for Image Generation

This code accompanies the paper:

[Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://arxiv.org/abs/2401.09603).  
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, Sanjiv Kumar.

The code has been tested with Python 3.11.4 on Linux and macOS.

## Instructions to Run

### 1. Setting things up

#### a. Install JAX

Follow the [JAX installation guide](https://jax.readthedocs.io/en/latest/installation.html)
to install the appropriate JAX distribution (with the correct GPU/TPU support) for your platform.

#### b. Set up Scenic

```
git clone https://github.com/google-research/scenic.git
export PYTHONPATH=$PYTHONPATH:/path/to/scenic
```

Scenic will automatically download CLIP model weights when you run the code as
described in the next section. If your network does not allow that, manually
download the ViT-L/14@336px CLIP model weights from [here](https://openaipublic.azureedge.net/clip/models/3035c92b350959924f9f00213499208652fc7ea050643e8b385c2dac08641f02/ViT-L-14-336px.pt)
and place them in `~/.cache/scenic/clip/`. See [here](https://github.com/openai/CLIP/blob/main/clip/clip.py#L30)
for more information on CLIP model weight download locations.


#### c. Install other dependencies

Other required libraries can be installed with the following command:

```
# from google-research/
pip install -r cmmd/requirements.txt
```


#### 2. CMMD calculation
Run the following command to calculate the CMMD between a reference set of images (such as COCO 30K images)
and a set of images to be evaluated (such as images generated by a model)
:

```
# from google-research/
python -m cmmd.main /path/to/reference/images /path/to/eval/images --batch_size=32 --max_count=30000
```

The two directories specified by the first two arguments are interchangeable and
should contain .jpg or .png images. The last two arguments (`batch_size` and
`max_count`) are optional.

For text-to-image evaluation, the two directories should contain real images for
a set of text prompts and generated images for the same prompts. We used the
[COCO 2014 val dataset](https://cocodataset.org/#download) for the experiments
presented in our paper. However, we have observed that this dataset is too low
in resolution to evaluate high-resolution image generation (e.g. 1024x1024).

---

Contact `sadeep {at} google.com` if you encounter any issues with this
code.
